User interface generation apparatus

ABSTRACT

A mobile phone includes a generation unit for generating a user interface for instructing an execution unit to execute a predetermined function based on an application program, a memory unit for storing a definition file including information defining an object of the user interface, and a selection screen generation unit for generating a screen to receive selection of the object. The selection screen generation unit generates the selection screen based on object definition information of a plurality of user interfaces instructed. The generation unit generates a definition file including definition information of selected objects and also generates a compound user interface based on the definition file.

CROSS REFERENCE TO RELATED APPLICATION

This is a Continuation of application Ser. No. 12/935,376 filed Sep. 29,2010, which in turn is a National Stage of Application No.PCT/JP2009/056199 filed Mar. 29, 2009, which claims the benefit ofJapanese Patent Application No. 2008-95558 filed on Apr. 1, 2008, theentire contents of which are incorporated herein by reference.

TECHNICAL FIELD

The present invention relates to user interface generation apparatuses,and particularly, to user interface generation apparatuses forgenerating user interfaces of mobile terminals.

BACKGROUND ART

The user interface (hereinafter, arbitrarily abbreviated to “UI”) of themobile terminal represented by a mobile phone has a great influence onoperability when a user operates the mobile terminal. Accordingly, theUI of the mobile terminal is one of important factors for the user topurchase the mobile terminal.

With significant multi-functioning of the mobile terminal in recentyears, there are numerous mobile terminals having picturesque UIs usinganimation and 3D display. In addition, there are mobile terminals havingUIs which can be customized as desired by users. Such mobile terminalsdramatically improve convenience for the users.

Conventional mobile terminals generally have UI designs and operationmethods which are different between manufacturers and models. That is,the UI is unique to each terminal and most of the conventional mobileterminals have UIs highly dependent on models. Therefore, when programcodes of another terminal are reused in developing a terminal, it leadsto voluminous change in parts of the UI to the program codes.Accordingly, it has been difficult to port program codes of a terminalto another terminal.

In order to deal with such a problem, a UI based on XML (ExtensibleMarkup Language) has been introduced. Expressing the UI by using an XMLfile whose description method is standardized facilitates reuse of theprogram codes and enables to port the UI to another model. Accordingly,it also enables to deal with UI alone independently from each terminal,such that the UI can be used by terminals of different models anddifferent manufactures.

Representatives of such XML-based UI are UI Foundation developed by TAT(http://www.tat.se/), VIVID UI developed by Acrodea, Inc.(http://www.acrodea.co.jp/), UI One developed by Qualcomm Incorporated(http://www.qualcomm.co.jp/) and the like.

In addition, with multi-functioning and high-performance of a terminalbody, an increasing number of recent mobile terminals mount OS(Operating System) which can perform the multi-task processing forsimultaneously executing a plurality of tasks in parallel. Moreover,mobile terminals with multi-window function to multiplex output screendisplays by allocating a plurality of tasks processed in parallel andsimultaneously to respective display areas (windows), have been becomingwidely used.

Incidentally, Japanese Patent Laid-Open No. 2001-36652 discloses ascheme of remote handling (control) of a plurality of externalequipments by infrared communication using a mobile phone terminalhaving an infrared communication unit. The mobile phone terminaldescribed in Japanese Patent Laid-Open No. 2001-36652 is provided withan external control unit for communicating with the external equipments.This mobile phone terminal stores the external equipment controlinformation for remotely controlling the external equipments, obtainedvia a telephone line or received from the external equipments, andremotely controls the external equipments based on the externalequipment control information. That is, by changing an applicationprogram (hereinafter, referred to as “application” simply) incorporatedin the terminal, the terminal body, which is normally used as the mobilephone, can be used as a remote controller (hereinafter, referred to as“remocon” simply) for remotely controlling the plurality of externalequipments.

SUMMARY OF INVENTION Technical Problem

According to the mobile phone terminal disclosed in the above JapanesePatent Laid-Open No. 2001-36652, a single mobile phone terminal canremotely control a plurality of external equipments based on respectiveexternal equipment control information corresponding to the externalequipments. Therefore, there is no need to do cumbersome operation for auser to separately use individual remocon terminals for the plurality ofexternal equipments, which improves convenience for the user.

If such functions of the remocon to remotely switch between theplurality of external equipments to be controlled remotely isimplemented in the mobile terminal capable of multi-task processingstated above, it is not necessary to finish a remocon application toinitiate another remocon application. That is, it is possible toinitiate a plurality of applications on a single terminal at the sametime and to use a desired remocon application among them timely.

However, the operation to switch the plurality of applicationsfrequently is cumbersome.

If the multi-windows function stated above is embedded in the mobileterminal, UIs for operating a plurality of applications can be displayedon a plurality of windows, respectively. Thereby, it is possible to usethe applications on a single screen simultaneously without switchingamong the plurality of applications.

For a TV remocon terminal alone as shown in FIG. 19(A), for example, UIof TV remocon application may be reproduced on a touch panel 200 of amobile terminal 100, for example, while maintaining operability of theTV remocon terminal, as shown in FIG. 19(B). Although keys such asnumerical keys or buttons of the remocon terminal shown in FIG. 19(A)are mechanical keys, the numerical keys or the buttons are reproduced asobjects and displayed on the touch panel 200 of the mobile terminal 100shown in FIG. 19(B). Similarly, for an air conditioner remocon terminalalone as shown in FIG. 19(C), UI of air conditioner remocon applicationmay be reproduced on the touch panel 200 of the mobile terminal 100,while maintaining operability of the air conditioner remocon terminal,as shown in FIG. 19(D). Moreover, if each UI of application is describedin XML as the basis, almost the same UI can be easily reproduced inanother terminal only by processing the XML file describing the UI, evenif the application is ported to the another terminal of a differentmanufacturer.

However, in order to operate an air conditioner while viewing a TV inthe above example, there may be a need for cumbersome operations forboth of the TV and the air conditioner. In such a case, the applicationsfor both the TV and the air conditioner may be displayed simultaneouslyby dividing a display area of the display unit by using the multi-taskfunction and the multi-window function. However, simultaneous display ofthe UIs results in displays as shown in FIG. 20(A) to (D), as eachapplication has only the UI which assumed to be used solely.

FIG. 20(A) to (D) are diagrams illustrating states when the UIs of twoapplications are simultaneously displayed on the mobile terminal 100having the multi-task function and the multi-window function. In theexamples shown in the figures, only a part of each UI is displayed, aseach of the UIs is reproduced as it stands in each of the windowsvertically separating the display unit of the touch panel 200. A scrollbar is provided on the right in each window so as to enable operation ofparts beyond the screen.

In order to adjust volume of the TV in a state shown as FIG. 20(A), forexample, the user needs to move the scroll bar in the window of the TVremocon UI so as to move a displayed area of the TV remocon UI, as shownin FIG. 20(B). Similarly, in order to adjust temperature of the airconditioner in a state shown as FIG. 20(C), for example, the user needsto move the scroll bar in the window of the air conditioner remocon UIso as to move a displayed area of the air conditioner remocon UI, asshown in FIG. 20(D). If a desired one by the user among the keys or thebuttons and information display portions (hereinafter, referred to as“UI object”), which are components of the UI, is not displayed in thedisplay area, the user needs to operate to move the display area everytime. Moreover, when a plurality of applications are activated by themulti-window function and the UIs are displayed overlapping each other,it is necessary to select the window to activate it before startinginput operation.

In particular, the terminal body of the mobile terminal is designed tobe compact, and thus an area which can be occupied by either UIconstituted of the touch panel or UI constituted of physical keys isvery limited. Therefore, an operation unit which constitutes the UI ofthe application such as the remocon is generally designed in a minimumsize. Accordingly, in order to display UIs of a plurality ofapplications, if each UI is reduced overall and displayed, each UIobject becomes too small, possibly deteriorating operability thereofsignificantly. That is, if the UI designed to occupy an entire screen isdisplayed in multi screens, it results in deterioration of operabilityof the UI.

In addition, the user using the terminal may not need all of the UIobjects composing the UI designed to occupy the entire screen. If UIobjects unnecessary for the user are included in the UI, such UI objectsare impeditive for operation and may possibly deteriorate overalloperability.

Therefore, it is an object of the present invention in consideration ofsuch conditions to provide a user an interface generation apparatuswhich compounds UIs only with selected UI objects and is capable ofmaintaining operability of each user interface, when user interfaces ofa plurality of application programs are reproduced simultaneously.

Solution to Problem

In order to achieve the above object, a user interface generationapparatus according to a first aspect includes:

an application program execution unit for implementing a variety offunctions based on an application program;

a user interface generation unit for generating a user interface forinstructing the application program execution unit to execute apredetermined function based on the application program;

a memory unit for storing a user interface definition file includinguser interface object definition information for defining a userinterface object, which is a component of the user interface; and

a selection screen generation unit for generating a selection screen forreceiving selection of the user interface object based on the userinterface object definition information, wherein

the selection screen generation unit, if it is instructed to compound aplurality of user interfaces, generates the selection screen based onthe user interface object definition information included in the userinterface definition file stored in the memory unit correspondingly toeach of the plurality of user interfaces instructed, and

the user interface generation unit generates a compound user interfacedefinition file including user interface object definition informationof user interface objects selected on the selection screen, andgenerates a compound user interface based on the compound user interfacedefinition file.

A second aspect of the present invention is that, in the user interfacegeneration apparatus according to the first aspect,

the user interface object definition information includes user interfaceobject attribute information including information indicating arelationship between the user interface object and another userinterface object composing the user interface including the userinterface object as a component, and the user interface generation unit,if it is instructed to compound the plurality of user interfaces,determines an arrangement of the user interface objects of the compounduser interface based on the user interface object attribute informationof the user interface objects selected on the selection screen andidentification information of user interfaces to which the userinterface objects belong.

A third aspect of the present invention is the user interface generationapparatus according to the second aspect, further including an occupancyaccumulation unit for accumulating occupancies of the user interfaceobjects, selected on the selection screen, in the compound userinterface based on the user interface object attribute information,wherein

the selection screen generation unit, if the occupancy of the compounduser interface accumulated by the occupancy accumulation unit exceeds apredetermined value, cancels selection of a user interface objectselected on the selection screen after the occupancy exceeds thepredetermined value and informs that the user interface object cannot beselected.

A fourth aspect of the present invention is that, in the user interfacegeneration apparatus according to any one of the first to the thirdaspects,

the memory unit stores the compound user interface definition filegenerated based on the user interface objects selected on the selectionscreen, and

the application program execution unit, when activating the compounduser interface based on the compound user interface definition filestored in the memory unit, activates a corresponding application programin the background based on relevant application program informationspecifying an application program corresponding to each user interfaceobject composing the compound user interface.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a schematic configuration of auser interface generation apparatus according to the present invention;

FIG. 2 is a table exemplifying relevant application program informationaccording to the present invention;

FIG. 3 is a flowchart illustrating overall processing to generate acompound user interface according to the present invention;

FIG. 4 is a diagram illustrating conditions to select user interfaces tocompound user interfaces according to the present invention;

FIG. 5 is a diagram illustrating conditions to select user interfaces tocompound user interfaces according to the present invention;

FIG. 6 is a diagram illustrating examples of a user interface objectselection screen displayed to compound user interfaces according to thepresent invention;

FIG. 7 is a diagram illustrating operations of the user interface objectselection screen according to the present invention;

FIG. 8 is a diagram illustrating operations of the user interface objectselection screen according to the present invention;

FIG. 9 is a diagram illustrating operations of the user interface objectselection screen according to the present invention;

FIG. 10 is a diagram illustrating operations of the user interfaceobject selection screen according to the present invention;

FIG. 11 is a diagram illustrating operations of the user interfaceobject selection screen according to the present invention;

FIG. 12 is a diagram illustrating operations of the user interfaceobject selection screen according to the present invention;

FIG. 13 is a diagram illustrating an example of a user interfacedefinition file according to the present invention;

FIG. 14 is a table exemplifying a user interface administration tablefor each application program according to the present invention;

FIG. 15 is a flowchart illustrating processing by a user interfacegeneration unit of the user interface generation apparatus according tothe present invention;

FIG. 16 is a flowchart further illustrating the user interfacegeneration processing by the user interface generation unit of the userinterface generation apparatus according to the present invention;

FIG. 17 is a flowchart illustrating overall processing performed whenthe user interface generation apparatus according to the presentinvention executes an application program alone;

FIG. 18 is a flowchart illustrating overall processing performed whenthe compound user interface according to the present invention executesthe application program;

FIG. 19 is a diagram illustrating states of conventional remoconterminals reproduced by the user interfaces of a mobile terminal; and

FIG. 20 is a diagram exemplifying compound of two user interfaces by aconventional mobile terminal.

REFERENCE SIGNS LIST

-   1 mobile phone-   10 control unit-   11 application program execution unit-   12 user interface obtaining unit-   13 user interface generation unit-   14 user interface object selection processing unit-   15 user interface resource shaping unit-   16 compound user interface definition file generation unit-   20 touch panel-   22 input unit-   24 display unit-   30 wireless communication unit-   40 infrared communication unit-   50 memory unit-   51 application program memory area-   52 user interface definition file memory area-   53 individual user interface resource memory area-   54 common user interface resource memory area-   55 relevant application program information memory area-   56 compound user interface definition file memory area

DESCRIPTION OF EMBODIMENTS

Embodiments of the present invention will be described with reference tothe accompanying drawings. In each of the embodiments described below, amobile phone having a remocon function for remotely controlling variousexternal apparatuses is used as an example of a user interface(hereinafter, referred to as “UI”) generation apparatus according to thepresent invention. However, the UI generation apparatus according to thepresent invention is not limited to the mobile phone but may beapplicable to any portable terminals such as, for example, a laptopcomputer, PDA and the like. In addition, the UI generation apparatusaccording to the present invention is also applicable to an apparatuswhich needs to mount it, even if it is not a mobile terminal. It is tobe noted that the present invention primarily intends to compound aplurality of UIs to be used simultaneously. Therefore, an applicationinstructed by each UI is not limited to a remocon function but thepresent invention may be applicable to various kinds of applications.

FIG. 1 is a block diagram illustrating a schematic configuration of a UIgeneration apparatus according to the embodiment of the presentinvention.

A mobile phone 1 includes a control unit 10 for controlling the overalloperation and a touch panel 20 for receiving input by a user and fordisplaying a result of input and the like, according to eachapplication. The touch panel 20 has an input unit 22 constituted ofmatrix switches and the like to receive input by the user and arrangedon top of a display unit 24 constituted of a liquid crystal display andthe like. The display unit 24 displays according to an applicationprogram and also displays an image of UI, such as various types of keysor buttons for receiving input by the user, in a predetermined UIdisplay area.

In addition, the mobile phone 1 also includes a wireless communicationunit 30 for transmitting and receiving various information such as voicecall, data of e-mail and the like to/from a base station (not shown) andan infrared communication unit 40 for communicating with variousexternal equipments (not shown) by using infrared rays. The mobile phone1 transmits and receives a variety of data to/from the externalequipments by using the wireless communication unit 30 via the internet,by radio and the like.

The mobile phone 1 further includes a memory unit 50 for storing inputinformation and various applications and also for functioning as a workmemory. The memory unit 50 includes an application memory area 51, a UIdefinition file memory area 52, an individual UI resource memory area53, a common UI resource memory area 54, a relevant applicationinformation memory area 55, and a compound UI definition file memoryarea 56.

The control unit 10 includes an application execution unit 11, a UIobtaining unit 12, a UI generation unit 13, a UI object selectionprocessing unit 14, a UI resource shaping unit 15, and a compound UIdefinition file generation unit 16.

The application execution unit 11 of the control unit 10 executesvarious applications stored in the application memory area 51 of thememory unit 50, as well as controlling such execution. In addition, theapplication execution unit 11, based on input to the UI corresponding tothe application stored in the application memory area 51, executesfunctions of the application corresponding to the input.

The UI obtaining unit 12 obtains resource (image data and the like) anda UI definition file (XML file and the like) from outside the terminalvia the wireless communication unit 30. The UI generation unit 13performs parsing process and DOM (Document Object Model) process on theUI definition file, and generates the UI to be used actually. The UIgeneration unit 13 interprets information of the UI, described in an XMLform, by using an XML engine and displays the UI generated based on theUI definition file on the display unit 24 of the touch panel 20.

By analyzing UI object definition information defining the UI object tobe shown on the display unit 24, the UI object selection processing unit14 controls and performs processing to generate a selection screen forreceiving selection of the UI objects by the user. Hereinafter, thescreen displaying the UI objects on the display unit 24 to receiveselection by the user is referred to as a “UI object selection screen”or simply “selection screen”. Accordingly, the UI object selectionprocessing unit 14 constitutes a selection screen generation unitaccording to the present embodiment. Moreover, the UI object selectionprocessing unit 14, when instructed to compound a plurality of UIs in apredetermined display area of the display unit 24, compounds the UIsbased on a plurality of specified UI definition files.

The UI object selection processing unit 14 includes an occupancyaccumulation unit (not shown) for accumulating occupancies of selectedUI objects on the UI object selection screen. The occupancy accumulationunit accumulates occupancies of the UI objects based on the UI objectattribute information included in the UI object definition informationof each UI object.

The UI resource shaping unit 15 performs preview processing of theselected UI object and processing to enlarge or reduce data of resourceused to compound UIs. In this case, data of resource stored in theindividual UI resource memory area 53 or the common UI resource memoryarea 54 are read out and used. The compound UI definition filegeneration unit 16 generates and outputs the UI compounded by the UIobject selection processing unit 14 as the compound UI definition file.

The application memory area 51 of the memory unit 50 stores a variety ofapplications. The UI definition file memory area 52 stores the UIdefinition file defining a sequence of generation rules to generate eachUI overall. The individual UI resource memory area 53 stores individualUI resource such as image data and character string (text) data used togenerate UI unique to each application. The common UI resource memoryarea 54 stores common UI resource such as image data and font datacommonly used by the UIs used by the terminal, except for the individualUI resource unique to each application. When the UI is actuallygenerated, the image data and the text data stored in the individual UIresource memory area 53 and the common UI resource memory area 54 aredisplayed on the display unit 24 of the touch panel 20.

Furthermore, the compound UI definition file memory are 56 stores the UIdefinition file generated by the compound UI definition file generationunit 16. In addition, the relevant application information memory area55 stores relevant application information including activationinformation of an application relevant to each compound UI definitionfile. FIG. 2 shows an example of the relevant application information.In FIG. 2, compound UI definition files are shown on the left andrelevant applications corresponding thereto are shown on the right.

Next, the UI definition file stored in the UI definition file memoryarea 52 according to the present embodiment is described.

The UI definition file memory area 52 of the memory unit 50 stores theUI definition file defining specifications and operations of anapplication of the UI, required to execute the application stored in theapplication memory area 51. Although the same UI may be used bydifferent applications, here it is assumed that the UI used by eachapplication differs from one another, for convenience of description,and that the UI definition files corresponding to the applications arestored. For example, a TV remocon UI definition file is stored in the UIdefinition file memory area 52, correspondingly to a TV remoconapplication for remotely controlling a TV set (not shown), which is theexternal equipment, by using the mobile phone 1. Similarly, an airconditioner remocon UI definition file is stored in the UI definitionfile memory area 52, correspondingly to an air conditioner remoconapplication for remotely controlling an air conditioner (not shown),which is another external equipment, by using the mobile phone 1.

In the present embodiment, a UIML (User Interface Markup Language) formbased on XML language is used as an example of a language to describethe UI definition file. In accordance with a definition described in theUIML form, the UI generation unit 13 displays the UI on the display unit24 of the touch panel 20 of the mobile phone 1 and the applicationexecution unit 11 executes processing in response to input to the inputunit 22 by the user.

The UI definition file according to the present embodiment includes UIobject definition information which defines a UI object to be shown onthe display unit 24. The UI object definition information includesinformation defining an image and a text for showing the UI object suchas key or button, which is a component of the UI to be displayed on thedisplay unit 24, and information defining an operation when there isinput to the UI object (in practice, when there is input to a part ofthe input unit 22 corresponding to the UI object) and the like.

According to the present embodiment, as the information defining theoperation when there is input to the input unit 22 corresponding to aposition of the UI object, action information, indicating contents ofthe operation to be executed by each application execution unit if anevent is occurred to each UI object, is included in the UI objectdefinition information. For example, if an input event occurs to theinput unit 22 corresponding to a position of the UI object of “Power” onthe TV remocon UI, the action information defines to issue aninstruction to TV remocon application to transmit an infrared signal toturn on or off the power of the TV as the external equipment.

As stated above, the UI object definition information includes the UIobject attribute object, which is information on each UI object used tocompound a plurality of UIs. A variety of information such asinformation on relationships with other UI objects composing the UI andinformation on priority to display each object based on the relationshipbetween each UI object and other UI objects when a plurality of UIs arecompounded may be added to the UI object attribute information. Inaddition, the UI definition file includes UI identification informationcorresponding to the UI definition file. A name of the UI, for example,may be added as text information to the UI identification information.

Next, a sequence of operations to generate the compound UI by the mobilephone 1 according to the present embodiment is described with referenceto FIG. 3 to FIG. 7.

According to the present embodiment, the user previously considers thatit is necessary to use a plurality of applications simultaneously butseparately, and selects UI objects from a plurality of the UI. Byadopting only the UI objects selected, the compound UI composed of onlythe UI objects desired by the user is generated. Therefore, the UIdefinition file memory area 52 of the memory unit 50 stores in advancethe UI definition file of each UI corresponding to the variety ofapplications stored in the application memory area 51.

FIG. 3 is a flowchart illustrating a sequence of overall processing togenerate the compound UI by the mobile phone 1 according to the presentembodiment.

First, based on input operation to the input unit 22 by the user and thelike, the application execution unit 11 reads out and activates anapplication for performing processing to generate the compound UI(compound UI generating application) from the application memory area 51(step S11). The compound UI generating application is configured to beable to perform processing to generate the compound UI by selecting onlydesired UI objects from the plurality of UIs.

When the compound UI generating application is initiated, the UI objectselection processing unit 14 displays conditions to narrow down UIdefinition file as a selection target on the display unit 24 such thatthe user designates a UI to select UI objects, from the plurality ofUIs, and receives specification of the conditions by the user at theinput unit 22 (step S12).

That is, when the compound UI generating application is activated, theUI object selection processing unit 14 displays a screen as show in FIG.4(A), for example, on the display unit 24 so as to receive specificationof the condition to select UIs for generating the compound UI. As shownin FIG. 4(A), when “Order of Obtained Time” is selected at the inputunit 22 by the user, for example, the UI object selection processingunit 14 displays names of the UIs of the UI definition file obtained bythe mobile phone 1 in reverse chronological order from an upper side ofthe screen, as shown in FIG. 4(B). The user may select a UI, whichincludes UI objects the user desired to adopt, among the names of theUIs listed. When listing up the names of the UIs on the display unit 24,the UI object selection processing unit 14 may display names in order ofnewly obtained by referring to information on time and date added (orupdated) when each UI definition file is stored in the UI definitionfile memory area 52 for the first time.

In addition, as shown in FIG. 5(A), if “Function”, for example, isselected on the input unit 22 by the user, the UI object selectionprocessing unit 14 displays respective UI names of UI definition filesby categorizing UIs according to function, that is, according toapplication (in this case, equipments corresponding to the remoconapplication are displayed), as shown in FIG. 5(B).

In addition to the conditions stated above, a variety of conditions maybe considered to select UIs to generate a compound UI, such as“Manufacturer” for categorizing the UIs according to manufacturers whichprovide respective UIs (or applications corresponding to the UIs),“Obtained spot” for categorizing the UIs according to spots (forexample, “My Room”, “Living Room” and the like) where the mobileterminal 1 obtains the UI definition files via the UI obtaining unit 12and the like. The following description is a case when the user selectsUI objects by categorization of “Function” of UIs, as shown in FIG.5(B).

When the user selects a condition as shown in FIG. 5(A) and UI names ofthe UI definition files are displayed based on the condition selected bythe user as shown in FIG. 5(B), the UI object selection processing unit14 analyzes contents of a plurality of UI definition files narrowed downby such selection and extracts the UI object definition informationincluded therein (step S13). After extracting the UI object definitioninformation of the UI objects, which may be adopted possibly, in thismanner, the UI object selection processing unit 14, when each UI isselected, displays information of UI objects composing the UI, based onthe UI object definition information (step S14).

That is, when each UI (for example, “Air Conditioner”, “TV” and thelike) whose name is shown in FIG. 5(B) is selected, text information(name of each key) of UI objects composing the UI is displayed as shownin FIG. 6(A) if the name of each object is given as the text informationto the UI definition file corresponding to each UI. In this case, inorder to display the names of the UI objects, the UI object selectionprocessing unit 14 extracts and displays text data associated with theUI objects included in the respective UI object definition information.FIG. 6(A) shows an exemplary case when “Air Conditioner” and “TV” amongthe UIs shown in FIG. 5(B) are selected (by input such as touchingcheckboxes and the like) and thereby UI objects composing the airconditioner remocon application and the TV remocon application aredisplayed.

When information of the UI objects of the selected UIs is displayed, theUI object selection processing unit 14 receives selection of the UIobjects to be adopted by the user. Therefore, according to the presentembodiment, the screen of the display unit 14 which displays theinformation on UI objects and receives selection by the user constitutesthe UI object selection screen.

When the information on the UI objects composing each UI is displayed asshown in FIG. 6(A), the user hardly selects only a “Temperature Up”(increase temperature) key without selecting a “Temperature Down”(decrease temperature) key of the UI of the air conditioner remoteapplication but usually selects both of the keys in pairs. Like this,the UIs used for remote applications have various UI objects to be usedin pairs. Therefore, it is preferred to allow the user to determinewhether to adopt such UI objects in pairs.

Therefore, relevance information indicating that a plurality of UIobjects are paired (or belong to the same group) is added as the UIobject attribute information to the UI object definition informationincluded in the UI definition file and the UI objects in pairs aredefined based on a value of the relevance information (attribute). Inexemplary UI definition files in an XML form shown at an end of thepresent specification, the relevance information of the UI object isshown as attribute (relate_ID). The plurality of UI objects may behandled as elements of the same group by adding the same value to theattribute (relate_ID).

As the UI objects in pairs are displayed in association with one anotherin this manner, it is possible to determine whether to adopt the UIobjects relevant to each other in pairs. In the example shown in FIG.6(B), the names of the UI objects “Temperature” and “Wind Direction” ofthe air conditioner remocon UI and the UI objects “Channel Selection”and “Volume” of the TV remocon UI are displayed in pairs and allowed todetermine whether to adopt or not in pairs. Collective display of thenames of such UI objects enables saving of a display space and effectiveutilization of a limited display area of the selection screen.

In addition, some UI definition file may not have all of names of the UIobjects composing each UI, as text information. In this case, that is,if text data which should be displayed are not contained in the UIobject definition information, the UI object selection processing unit14 reads out image data of the UI object from resource data stored inthe individual UI resource memory area 53 or the common UI resourcememory area 54 correspondingly to each UI and displays the image in areduced size.

Thereby, even if there is no text information of a name of a UI object,it is possible to display the UI object composing the UI to be visuallyrecognizable by the user, as shown in FIG. 6(C). FIG. 6(C) shows anexample that, since there is no text information corresponding to the UIobjects of “Display Window” and “Direct Channel Selection Key” in the UIobject definition information, these UI objects are displayed beingreduced.

It is also possible to display reduced images of all UI objects asdefault setting of the mobile phone 1 or as setting desired by the user,as shown in FIG. 6(D). Since each UI object certainly uses an image whendisplayed on the display unit 24 to be used as the UI, the image of theUI object in the reduced size may be displayed regardless of whether theUI object includes the text information of the name. Thus, display ofnot only the text information but also the images facilitates visualselection of the UI object of the user. In an example shown in FIG. 6(D)as well, the UI objects which should be handled as pairs arecollectively displayed by using the relevance information of the UIobjects.

After displaying the UI object selection screen shown in one of FIG.6(A) to (D), the UI object selection processing unit 14 receivesselection input of the UI object by the user (step S15). On receivingselection input (by input such as touching the checkboxes and the like)at the input unit 22 by the user (Yes of step S15), the UI objectselection processing unit 14 calculates a total of ratios of areas(hereinafter, referred to as “accumulated screen occupancy”) occupied bythe selected UI objects in a predetermined UI display area of thedisplay unit 24 (step S16). For this processing, an occupancy of each UIobject in the UI display area is included in advance as the UI objectattribute information in the UI object definition information of each UIobject.

At this time, if the UI objects are selected as shown in FIG. 7(B) onthe UI object selection screen shown in FIG. 7(A), for example, it ispreferred to display a bar chart and the like of the accumulated screenoccupancy at a lower section of the UI object selection screen excludinga soft key area at a bottom of the display unit 24. In an example shownin FIG. 7(B), UI objects “Power” (occupancy 10%) and “Display Window”(occupancy 30%) are selected from the air conditioner remocon UI and anUI object “Power” (occupancy 10%) is selected from the TV remocon UI.Therefore, the bar chart at the lower section in the selection screenindicates that the accumulated screen occupancy is 50%. This means thatthe UI objects which occupy 50% of the UI display area when the UI isactually compounded and displayed on the display unit 24, are currentlyselected. Blacked checkboxes in FIG. 7 indicate that the UI objects withsuch checkboxes are selected.

FIG. 7(C) shows that the UI object “Temperature Up/Down” (occupancy 15%)is selected from the air conditioner remocon UI by further input to theinput unit 22 by the user and the accumulated screen occupancy isincreased to 65%. Although input by the user to sequentially add the UIobject is received, the compound UI cannot be displayed in the UIdisplay area if the accumulated screen occupancy exceeds a predeterminedthreshold (for example, 100%). Therefore, the UI object selectionprocessing unit 14 checks whether the accumulated screen occupancyexceeds the predetermined threshold (step S17).

Since other UI objects can be added in a manner stated above so far asthe accumulated screen occupancy does not exceed the predeterminedthreshold (No of step S17), the UI object selection processing unit 14returns to step S15 to receive further input to the input unit 22 by theuser. However, if the accumulated screen occupancy exceeds thepredetermined threshold when the UI object is selected (Yes of stepS17), this UI object cannot be arranged. Thus, the UI object selectionprocessing unit 14 displays a warning message on the display unit 24 asshown in FIG. 7(D), for example (step S18).

Instead of using the warning message at step S18 or together with thewarning message, it is also possible to show the UI objects which, ifselected, make the accumulated screen occupancy exceed the predeterminedthreshold, in gray-out to be unselectable. In the selection screen shownin FIG. 7(D), the UI objects which, if selected, make the accumulatedscreen occupancy exceed 100% are shown in gray-out to indicate that theycannot be selected, and the warning message is displayed as the userinputs to select one of these objects regardless of such condition.

Since the UI object selected beyond the predetermined threshold of theaccumulated screen occupancy cannot be arranged when composing the UI,the UI object selection processing unit 14 cancels selection of the UIobject selected beyond the threshold (step S19) and returns to step S15to receive selection of another UI object (which does not make theaccumulated screen occupancy exceed the predetermined threshold).

In this manner, the user can select the UI objects from the plurality ofUIs as desired, so far as the accumulated screen occupancy does notexceed the predetermined threshold. In addition, when the user selectsthe UI objects, it may be difficult for the user to recognize actualarrangement of the UI objects with the bar chart of the accumulatedscreen occupancy alone. In such a case, in response to input to the softkey “Preview” at bottom right corner of the display unit 24 as shown inFIG. 8(A), for example, the UI object selection processing unit 14 maydisplay a preview screen of current arrangement of the UI objects. Suchpreview screen enables the user to determine whether to adopt the UIobjects while checking actual arrangement of the UI objects.

In addition, there may be a case in which, when determining whether toadopt a UI object, the user wishes to slightly reduce UI objects alreadyselected, so as to further add another UI object. According to thepresent embodiment, therefore, the UI object selection screen may bedisplayed as shown in FIG. 9(A), such that the UI objects can bereduced. In FIG. 9(A), the occupancy of the UI object “Power” of the airconditioner remocon UI displayed as 8[5-15] means that the occupancy ofthe UI object is 8% currently but may be adjusted between 5% and 15%. InFIG. 9(A), all UI objects are reduced to have smaller occupancies incomparison with those in FIG. 8. If the preview display described aboveis executed at FIG. 9(A), UI objects smaller than those in FIG. 8(B) andthe reduced bar chart indicating the accumulated screen occupancy aredisplayed as shown in FIG. 9(B). It is possible to reduce all of theplurality of UI objects currently selected collectively or to reduceeach UI object individually.

Processing to enlarge or reduce the UI object is performed by the UIresource shaping unit 15 when the UI object selection processing unit 14reads out each UI object from the individual UI resource memory area 53or the common UI resource memory area 54.

Also, in order to reduce the UI objects as stated above, it is preferredto set an upper limit and a lower limit of the occupancy inconsideration of a practical size of each UI object. Both of the limitsare set based on a type of the UI object and a resource to be used, andstored as the UI object definition information for each UI object. Itthereby prevents to generate a user-unfriendly UI with the UI objectsreduced too much.

In addition, there may be a case in which operations to switch betweenthe preview screen with the UI object arranged and the UI objectselection screen are burdensome. Therefore, according to the presentembodiment, it is possible to add the UI object while the preview screenis displayed.

That is, in response to input by the user to a soft key “Add UI” atbottom right corner on the display unit 24 while the preview screen isdisplayed as shown in FIG. 10(A), a UI object adding screen is displayedas shown in FIG. 10(B). In an example shown in FIG. 10(B), UI objectsnot selected yet are displayed for each UI so as to receive selectioninput by the user. In the example shown in FIG. 10, the UI objects areclassified by tab display, for each UI of application. On the tab shownin FIG. 10(B), “A/C” indicates the air conditioner.

In this case, it is preferred that each UI object can be dragged inaccordance with input to the touch panel 20 with the user's finger and,as shown from FIG. 10(C) to (D) in series, that the UI object isselected in response to input by dragging upward. Occupancy of the UIobject selected in this manner is reflected in the bar chart. An areanecessary to arrange each UI object is shown by dashed lines surroundingthe UI object. Providing a reasonable space around the image of the UIobject enables to maintain certain operatively even if multiple UIobjects are arranged adjacently.

In the UI object adding screen, the user may arrange the UI objects asdesired by dragging each UI object. As shown in FIG. 10(D) to FIG.10(E), a position of the UI object is determined when the user detacheshis/her finger from the touch panel 20 after positioning the UI object.

In addition, there may be UI objects which are not in a relationship ofequals but in a master/subordinate relationship with one another. Thatis, it is the first operation for most of remocons to turn power on,followed by other operations. In case of the TV remocon UI, for example,there is no necessity to select a channel or to adjust volume withoutturning power on. In this case, accordingly, it may be considered thatan object of power is “master” and other objects in association with thefollowing operations to the power ON are “subordinate”. According to thepresent embodiment, when the user selects UI objects, the user may notbe allowed to select “subordinate” UI objects before selecting a“master” UI object.

As shown in FIG. 11(A), for example, UI objects subordinate to the UIobject “Power” in each UI are grayed out to be unselectable before theUI object “Power” is selected. When the master UI object “Power” isselected, it is allowed to select the UI objects subordinate to themaster UI object as shown in FIG. 11(B).

In order to add the master/subordinate relationship as stated above tothe UI objects, the UI object attribute information indicating that aplurality of UI objects are in the master/subordinate relationship isadded to the UI object definition information included in the UIdefinition file. Thereby, it is possible to reduce possibility that theuser forgets to select a necessary UI object or that the user selectsunnecessary UI object by an incorrect operation.

When receiving selection of the UI objects as stated above, if the userdoes not select a UI object (No of step S15), the UI object selectionprocessing unit 14 determines whether the operation to select the UIobjects is finished (step S20). For example, if there is input to a softkey “OK” on the selection screen shown in FIG. 12(A) or on the previewscreen shown in FIG. 12(B) by the user, it is determined that theoperation to select the UI objects is finished. If the selectingoperation of the UI objects is not finished at step S20, the UI objectselection processing unit 14 returns to step S15 to receive furtherinput to the input unit 22 by the user.

If the operation to select the UI objects is finished at step S20, theUI object selection processing unit 14 performs automatic layoutprocessing with selected UI objects (step S21). If there is noinstruction by the user, the UI objects are automatically arranged fromtop left in order of selection, in principle, so far as the UI displayarea permits. For example, if a plurality of UI objects is selected inseries until the accumulated screen occupancy reaches 100% as shown inFIG. 12(A), the UI objects are arranged to fit in the UI display area ofthe display unit 24 in order of selection, in principle, as shown inFIG. 12(B). It is preferred not to left-align the UI objects simply butto arrange relevant UI objects collectively based on the UI objectdefinition information and the like.

Even after automatic arrangement of the UI object, the user may suitablycheck current arrangement of the UI objects on the preview screen, asstated above. At this time also, arrangement of the UI objects can bechanged in accordance with an operation to drag the input unit 22 by theuser.

When layout of the selected UI objects is completed, the compound UIdefinition file generation unit 16 generates the compound UI definitionfile to define a plurality of compound UIs composed of the selected UIobjects, based on the UI object definition information of the selectedUI objects (step S22). The compound UI definition file generated isstored as a file in the XML form in the compound UI definition filememory area 56 of the memory unit 50. The file may be formed like the UIdefinition file of the XML form shown at the end of the presentspecification.

When the compound UI definition file is generated, the compound UIdefinition file generation unit 16 generates the relevant applicationinformation based on the selected UI objects of the compound UI (stepS23). The relevant application information generated here is informationto control applications relevant to UI objects when there is input tothe UI objects selected for the compound UI. When the relevantapplication information is generated based on the selected UI objects,the compound UI definition file generation unit 16 associates therelevant application information with the compound UI definition fileand stores the information in the relevant application informationmemory area 55 (step S24), and then ends overall processing o generatethe compound UI.

Thereafter, it is possible to immediately use the compound UI oncegenerated, by reading out the compound UI definition file stored in thecompound UI definition file memory area 56.

The following is a method to generate the relevant applicationinformation corresponding to the compound UI definition file composed ofthe selected UI objects.

A first method to generate the relevant application information collectsand uses information on applications included in respective UIdefinition files. That is, as shown in FIG. 13, the UI definition fileincludes information on application (application_(—)used=“tvremoconapp”) which uses a UI generated based on the UIdefinition file. The control unit 10 collects information on theapplications, which use the UIs generated based on the UI definitionfiles, respectively, from the UI definition files of the UIs having theUI objects, which are selected as components of the compound UI, andgenerates relevant application information 55 a of the compound UI, asshown in FIG. 2. The generated relevant application information 55 a isassociated with a corresponding compound UI definition file and thenstored in the relevant application information store area 55.

A second method to generate the relevant application informationcollects and uses information on relevant applications from a list(table) stored in advance for administration of the UI for eachapplication. That is, as shown in FIG. 14, a UI administration table foreach application 57, in which each UI definition file and information ofan application which uses the UI generated by the UI definition file areassociated with each other, is stored in the memory unit 50 in advance.With reference to the UI administration table for each application 57,the control unit 10 collects the information on the applicationscorresponding to the UI definition files of the UIs, having the UIobjects selected as components of the compound UI, as components andgenerates the relevant application information 55 a of the compound UIas shown in FIG. 2. In this case also, the relevant applicationinformation 55 a is associated with a corresponding compound UIdefinition file and then stored in the relevant application informationmemory area 55.

In addition, there may be a case in which a single application has aplurality of UIs. For example, “Calculator Application” has a normalcalculator UI to which the most basic keys (UI object) for basiccalculation, such as four arithmetic operations, are assigned and afunctional calculator UI to which keys (UI object) for science andtechnology calculation, such as trigonometric function and power, areassigned. As these two UIs are provided switchably therebetween, it ispossible to operate functions of the calculator application inaccordance with the user's purpose. For example, in the UIadministration table for each application 57 shown in FIG. 14, eachinformation on application (application name) is associated with a typeof the UI of the application (UI_code) and the UI definition file (theUI definition file name) of the UI. Here, the calculator application“calculatorapp” is associated with a normal calculator UI“Simple_Cal_UI” and a functional calculator UI “Function_Cal_UI” basedon different UI definition files (“Simple_calculator_interface.uiml” and“Function_calculator_interface.uiml”), respectively.

According to the present embodiment, as stated above, since it ispossible to generate a single compound UI by adopting only the UIobjects selected from the plurality of UIs, the compound UI composed ofonly UI objects desired by the user can be provided. FIG. 12(B) is anexemplary diagram of a state in which layout of the compound UI iscompleted. In this example, a single compound UI is generated havingonly UI objects, necessary for the user in accordance with user'sselection, from three kinds of UIs: the air conditioner remocon UI, theTV remocon UI and a lighting A UI.

Although overall processing to generate the compound UI is described asset forth above, in order to initiate and actually use the compound UIgenerated (prepared) as the compound UI definition file, it is necessaryto display each UI object as a single compound UI on the display unit 24and set the UI enabled. The following is a description of overalloperation of the UI generation unit 13 when the mobile phone 1 displaysand enables a normal UI or the compound UI according to the presentembodiment.

FIG. 15 is a flowchart illustrating overall processing of the UIgeneration unit 13 to generate the UI from the UI definition file or thecompound UI definition file and enable the UI. First, when the UIdefinition file (XML file) is specified from an application authorizedto display the UI, based on input operation to the input unit 22 and thelike by the user, processing to generate and enable the UI is started.Such application authorized to display the UI is not activated in thebackground of the mobile phone 1 but operating at the top of theapplications.

Next, the UI generation unit 13 reads out specified UI definition filefrom the UI definition file memory area 52 or the compound UI definitionfile memory area 56 and determines whether the UI definition file is thecompound UI definition file (step S31).

If it is determined that the specified file is the compound UIdefinition file, the UI generation unit 13 obtains the relevantapplication information stored in association with the target compoundUI definition file from the relevant application information memory area55 of the memory unit 50 (step S32). Next, based on the relevantapplication information obtained, the UI generation unit 13 instructsthe application execution unit 11 to activate relevant applications inthe background (step S33).

Then, the UI generation unit 13 performs XML parsing process (step S34)and DOM process (step S35) on the specified compound UI definition file.In addition, the UI generation unit 13 performs a UI generationprocessing based on the UI definition file on which parsing process andDOM process are performed (step S36). The UI generation processing isdescribed below. Next, the UI generation unit 13 performs processing todisplay the UI in the UI display area of the display unit 24 of thetouch panel 20 based on the UI of the XML file generated by the UIgeneration processing (step S37), and thereby the UI is enabled.Accordingly, the UI generation unit 13 and the compound UI definitionfile generation unit 16 together constitute a user interface generationunit in the present embodiment.

If it is determined that the specified file is not the compound UIdefinition file at step S31, the UI generation unit 13 returns to stepS34 and thereafter performs the same processing as stated above. In thiscase, the UI definition file to be processed is not the compound UIdefinition file but the UI definition file for a single UI read out fromthe UI definition file memory area 52.

Next, the UI generation processing at step S36 in FIG. 15 is furtherdescribed with reference to a flowchart shown in FIG. 16.

At start of the UI generation processing shown in FIG. 16, either the UIbased on the single UI definition file or the compound UI based on theplurality of UI definition files is formed as a single DOM document (orDOM node). First, the UI generation unit 13 performs analysis inrelation to display of the DOM document (step S51). For the UIdefinition file described in the UIML form, attributes with structuretag and style tag are analyzed. Next, the UI generation unit 13 performsanalysis in relation to operation of the DOM document (step S52). Forthe UI definition file described in the UIML form, attribute withbehavior tag and the like are analyzed.

Next, the UI generation unit 13 performs conversion processing of anexpression included in the DOM document into a description dependent oneach terminal (step S53). Moreover, the UI generation unit 13 selects aresource of the UI object to be used based on a result of the conversionprocessing at step S53 and sets each attribute (property) (step S54).The resource of the UI object necessary at this time is stored as theindividual UI resource in the individual UI resource memory area 53 oras the common UI resource in the common UI resource memory area 54.Subsequently, the UI generation unit 13 performs plot processing of theUI including each UI object (step S55). Thereby, the UI generationprocessing ends and the UI generation unit 13 shifts to step S37 in FIG.15 for UI display processing.

Thereby, it is possible to perform processing accordingly when the UIdefinition file specified in UI generation is either the compound UIdefinition file to compound a plurality of UIs or the UI definition fileof the single UI, which does not compound a plurality of UIs. That is,if the compound UI definition file is processed, the application tocompound a plurality of UIs is prepared and the compound UI isgenerated, whereas a single UI is generated in a conventional manner ifthe UI definition file of the single UI is processed.

Although the overall processing to generate and enable the UI by the UIgeneration unit 13 is described above, in order to actually initiate anduse the compound UI generated in this manner, it is necessary to enablerespective applications, corresponding to the UIs simultaneously used,simultaneously. Therefore, after reading out the compound UI definitionfile in the XML form from the compound UI definition file memory area 56and initiating it, it is necessary to perform processing such asactivation of each application, relevant to the compound UI, in thebackground.

For description of such processing, the following is a description ofoverall processing performed when there is only one application whichreceives instruction from the UI and such specified application isexecuted alone, with reference to a flowchart shown in FIG. 17.

When an application desired to initiate is specified by the user withinput to the input unit 22 on a menu screen and the like, theapplication′execution unit 11 reads out such specified application fromthe application memory area 51 and initiate it (step S71). Next, theapplication execution unit 11 specifies the UI definition file definingthe UI, which is first used by the application activated (hereinafter,referred to as “current application” appropriately), to the UIgeneration unit 13 (step S72). The UI generation unit 13 reads out suchspecified UI definition file from the UI definition file memory unit 52,generates the UI based on the UI definition file read out and displaysthe UI on the display unit 24 (step S73). This processing to generateand enable the UI is performed based on the processing in the abovedescription with reference to FIG. 15 and FIG. 16.

Then, the application execution unit 11 determines whether an event suchas input operation to the input unit 22 by the user has been occurred tothe UI displayed. If there is an event (Yes of step S74), theapplication execution unit 11 processes the event by the UI currentlydisplayed (step S75). Subsequently, the application execution unit 11determines whether processing of the event is completed only with the UIcurrently displayed (step S76). If the processing of the event iscompleted only with the UI currently displayed, the applicationexecution unit 11 returns to step S74 to continue processing. However,if the event is not completed only with the UI currently displayed (Noof step S76), the application execution unit 11 processes the event byusing functions of the application (step S77).

For the process using the functions of the application at step S77, theapplication execution unit 11 determines whether there is a necessity toswitch to another UI (step S78). If there is no such necessity, theapplication execution unit 11 determines whether there is a necessity toinitiate another application (step S79). If there is no such necessity,the application execution unit 11 then determines whether there is anecessity to end the current application (step S80). If there is no suchnecessity, the application execution unit 11 returns to step S74 tocontinue processing. However, if it is determined that there is thenecessity to end the current application at step S80, the applicationexecution unit 11 performs ending process of the application (step S81).The ending process of the application includes processing to discard theUI currently displayed.

On the other hand, if it is determined that there is the necessity toswitch to another UI, the application execution unit 11 discards the UIcurrently displayed and determines a next necessary UI for the currentapplication. When the necessary UI is determined, the applicationexecution unit 11 reads out the UI definition file defining thenecessary UI from the UI definition file memory area 52 and specifies itto the UI generation unit 13 (step S82). Based on such specification,the UI generation unit 13 generates a new UI and displays it on thedisplay unit 24 (step S83). Such generation of the new UI is alsoperformed based on the processing described with reference to FIG. 15and FIG. 16. After generating and displaying the new UI, the UIgeneration unit 13 returns to step S74 to continue processing.

In addition, if it is determined that there is the necessity to initiateanother application at step S79, the application execution unit 11determines and executes another application to be executed next (stepS84). Moreover, the application execution unit 11, while executing suchanother application, shifts the current application into a suspend state(step S85).

Thereby, if only one application receives instruction from the UI, it ispossible to continue operation of application even if an event acrossanother UI or another application occurs, by shifting processingappropriately.

Next, overall processing when a plurality of applications receivesinstruction from the compound UI, and a plurality of applicationsspecified is executed, is described with reference to a flowchart shownin FIG. 18. The overall processing shown in FIG. 18 includes similarprocessing to that shown in FIG. 17, thus a description thereof isomitted appropriately.

First, the application execution unit 11 initiates an application(compound UI execution application) for executing the UI based on thecompound UI definition file generated by the processing described withreference to FIG. 3 (step S91). Next, the application execution unit 11specifies the compound UI definition file to be used by the compound UIexecution application to the UI generation unit 13 (step S92). The UIgeneration unit 13 reads out such specified compound UI definition filefrom the compound UI definition file memory area 56 and, based on thecompound UI definition file read out, generates the compound UI anddisplays it on the display unit 24 (step S93). While displaying thecompound UI, the application execution unit 11 activates applicationsrelevant to the compound UI in the background based on the relevantapplication information read out from the relevant applicationinformation memory area 55 (step S94).

Subsequently, the application execution unit 11 determines whether anevent such as input operation to the input unit 22 by the user has beenoccurred on the compound UI displayed. If such event has been occurred(Yes of step S95), the application execution unit 11 processes the eventwith the compound UI currently displayed (step S96). After this, theapplication execution unit 11 determines whether the processing of theevent is completed only with the compound UI currently displayed (stepS97). If the event is completed only with the compound UI currentlydisplayed, the application execution unit 11 returns to step S95 tocontinue processing. However, if the event is not completed only withthe current compound UI (No of step S97), the application execution unit11 specifies an application operated in the background, and relevant tothe UI object operated by the event (step S98). Then, the applicationexecution unit 11 performs event processing by using functions of thespecified application (step S99).

If the event processing is performed by using the functions of theapplication at step S99, the application execution unit 11 determineswhether there is a necessity to switch to another UI (step S100). Ifthere is no such necessity, the application execution unit 11 determineswhether there is a necessity to initiate another application (stepS101). If there is no such necessity, the application execution unit 11then determines whether there is a necessity to end the currentapplication (step S102). If there is no such necessity, the applicationexecution unit 11 returns to step S95 to continue processing. However,if it is determined that there is the necessity to end the currentapplication at step S102, the application execution unit 11 performsending process of the applications operated in the background andrelevant to the compound UI being displayed (step S103) and thenperforms ending process of the current application (step S104). Thisending process of the applications includes processing to discard the UIcurrently displayed. Other processing are the same as those in thedescription with reference to FIG. 17.

Thereby, when there is a plurality of applications which receivesinstructions from the compound UI, it is possible to continue operationof application by appropriate processing regardless of an event acrossanother UI or another application.

It is to be understood that the present invention is not limited to theembodiments set forth above but may be modified or varied in a multiplemanner. For example, the mobile terminal having the touch panel executesthe UI compound processing in the above embodiment. However, the touchpanel is not an essential element for the UI compound processingaccording to the present invention. The present invention may beapplicable to a terminal with any input unit, such as numerousmechanical keys or a pointing device, for example, with which a user maycompound UIs.

In addition, although the mobile phone 1 remotely controls the externalequipments with infrared communication by the infrared communicationunit 40 in the above embodiment, it is also possible to adopt Near FieldCommunication such as Bluetooth (registered trademark), wireless LAN andthe like, for example, for communication with the external equipments.

Moreover, although the UI definition file of the UI corresponding toeach application is stored in the UI definition file memory area 52 inadvance in the above embodiment, it is also possible to obtain anecessary UI definition file externally as necessary. In this case, ifthe UI definition file is specified from the application using the UI,it is determined whether the necessary UI definition file is stored inthe UI definition file memory area 52 or the compound UI definition filememory are 56. If the necessary UI definition file is not stored in theUI definition file memory area 52 or the compound UI definition filememory are 56, the UI obtaining unit 12 of the control unit 10 obtainsthe necessary UI definition file from external equipment or an externalserver (not shown) via the wireless communication unit 30.

Shown below are exemplary files in the UIML form based on XML asexamples of the UI definition file used in each of the aboveembodiments. Underlined parts show additional UI object attributeinformation according to the present invention. In the examples, the UIobject is defined by <template> tag, and thus a description between<template> and </template> corresponds to the UI object definitioninformation.

First, an exemplary TV1_interface.uiml file is shown as the UIdefinition file constituting the TV remocon UI.

<?xml version =″1.0″ encoding=″Shift_JIS″?> <uiml>      <interface id =″TV1_interface″ particular name = ″TV1″>      <structure>         <partclass=″G:TopContainer″ id=″top″>         <part class=″G:Image″ id=″bg″/>        <part class=″VBOX″ id=″vbox1″>          <part class=″G:Area″id=″a1″ source=″#t1_switch1″ how=″replace″/>          <partclass=″G:Area″ id=″a6″ source=″#t1_10key″ how=″replace″/>          <partclass=″HBOX″ id=″hbox1″>           <part class=″G:Area″ id=″a2″source=″#t1_select_up″ how=″replace″/>           <part class=″G:Area″id=″a3″ source=″#t1_select_down″ how=″replace″/>          </part>         <part class=″HBOX″ id=″hbox2″>           <part class=″G:Area″id=″a4″ source=″#t1_volume_up″ how=″replace″/>           <partclass=″G:Area″ id=″a5″ source=″#t1_volume_down″ how=″replace″/>         </part>         </part>         </part>      </structure>     <style>         <property part-name=″bg″name=″image-src″>TV_resorce_bg.jpg</property>         <propertypart-name=″header″ name=″g:location″>0,0</property>         <propertypart-name=″header″ name=″g:size″>240,400</property>      </style>     <behavior>      </behavior>   </interface>   <template id =″t1_switch1″ priority = ″0″>      <part>         <part class=″G:Button″id=″switch1″/>      <style>      <propertyname=″image-src″>TV_resorce_switch1.jpg</property>      <propertyname=″g:text″>Power</property>        <propertyname=″g:size″>30,30</property>    </style>    <behavior>       <rule>       <condition>        <op name=″and″>          <eventclass=″KeyListener.g:keypressed″/>          <op name=″equal″>          <property event-class=″KeyListener.g:keypressed″name=″keyCode″ />           <constant value=″57398″ />          </op>       </op>     </condition>     <action>            <callname=″TVApp.on″/>     </action>        </rule>    </behavior>    </part>  </template>  <template id = ″t1_select up″priority = ″1″ relate_id=″select_1″>     <part>        <partclass=″G:Button″ id=″select_up″/>     <style>     <propertyname=″image-src″>TV_resorce_select_up.jpg</property>     <propertyname=″g:text″>Channel Selection +</property>     <property name=″g:size″>30,20</property>    </style>    <behavior>     <rule>       <condition>        <op name=″and″>          <eventclass=″KeyListener.g:keypressed″/>          <op name=″equal″>          <property event-class=″KeyListener.g:keypressed″name=″keyCode″ />           <constant value=″57399″ />          </op>       </op>        </condition>        <action>          <callname=″TVApp.select_up″/>        </action>    </rule>  </behavior>  </part> </template> <template id = ″t1_select_down″priority = ″1″relate_id=″select_1″>   <part>    <part class=″G:Button″id=″select_down″/>   <style>   <propertyname=″image-src″>TV_resorce_select_down.jpg</property>   <propertyname=″g:size″>30,20</property>   <property name=″g:text″>ChannelSelection −</property>  </style>  <behavior>   <rule>    <condition>   <op name=″and″>     <event class=″KeyListener.g:keypressed″/>     <opname=″equal″>      <property event-class=″KeyListener.g:keypressed″name=″keyCode″ />      <constant value=″ 57340″ />     </op>    </op>   </condition>    <action>     <call name=″TVApp.select_down″/>   </action>    </rule>  </behavior>   </part> </template> <template id= ″t1_volume_up″ priority = ″1″ relate_id=″volume_1″>   <part>    <partclass=″G:Button″ id=″volume_up″/>   <style>   <propertyname=″image-src″>TV_resorce_volume_up.jpg</property>   <propertyname=″g:size″>20,30</property>   <property name=″g:text″>Volume+</property>  </style>  <behavior>   <rule>    <condition>    <opname=″and″>     <event class=″KeyListener.g:keypressed″/>     <opname=″equal″>      <property event-class=″KeyListener.g:keypressed″name=″keyCode″ />      <constant value=″57341″ />     </op>    </op>   </condition>    <action>     <call name=″TVApp.volume_up″/>   </action>    </rule>  </behavior>   </part> </template> <template id= ″t1_volume_down″ priority = ″1″ relate_id=″volume_1″>   <part>   <part class=″G:Button″ id=″volume_down″/>   <style>   <propertyname=″image-src″>TV_resorce_volume_down.jpg</property>   <propertyname=″g:size″>20,30</property>   <property name=″g:text″>Volume−</property>  </style>  <behavior>   <rule>     <condition>        <opname=″and″>         <event class=″KeyListener.g:keypressed″/>        <op name=″equal″>           <propertyevent-class=″KeyListener.g:keypressed″ name=″keyCode″ />          <constant value=″57342″ />         </op>        </op>    </condition>     <action>        <call name=″TVApp.volume_down″/>    </action>     </rule>      </behavior>       </part>    </template>   <template id = ″t1_10key″ priority = ″3″>       <part>         <partclass=″G:Image″ id=″10key″/>       <style>       <propertyname=″image-src″>TV_resorce_10key.jpg</property>       <propertyname=″g:text″>Direct Channel Selection Key</property>      </style>     <behavior>       <rule>         <condition>          <eventclass=″KeyListener.g:keypressed″/>        </condition>         <action>        <call name=″TVApp.receiveKey″>          <param>         <property event-class=″KeyListener.g:keypressed″name=″keyCode″/>          </param>         </call>         </action>        </rule>      </behavior>       </part>    </template>    <peers>      <presentation base=″XXX.uiml″ />       <logic id =″log1″ source =″#t1_logic″>       </logic>    </peers> </uiml>

Next, the following is an exemplary AIR1_interface.uiml file as the UIdefinition file constituting the air conditioner remocon UI.

<?xmlversion=″1.0″ encoding=″Shift_JIS″?> <uiml>   <interface id =″AIR1_interface″ particular name = ″Air Conditioner Manufactured by X″>    <structure>        <part class=″G:TopContainer″ id=″top″>        <part class=″G:Area″ id=″a1″ source=″#t1_switch2″how=″replace″/>         <part class=″G:Area″ id=″a2″source=″#t1_wind_up″ how=″replace″/>         <part class=″G:Area″id=″a3″ source=″#t1_wind_down″ how=''replace″/>         <partclass=″G:Area″ id=″a4″ source=″#t1_set_up″ how=″replace″/>         <partclass=″G:Area″ id=″a5″ source=″#t1_set_down″ how″=replace″/>        <part class=″G:Area″ id=″a6″ source=″#t1_set_window″how=″replace″/>         <part class=″G:Area″ id=″a7″source=″#t1_mode_change″ how=″replace″/>         <part class=″G:Area″id=″a8″ source=″#t1_menu″ how=″replace″/>         <part class=″G:Image″id=″bg″/>        </part>     </structure>     <style>        <propertypart-name=″bg″ name=″image-src″>AIR1_resorce_bg.jpg</property>       <property part-name=″header″ name=″g:location″>0,0</property>       <property part-name=″header″ name=″g:size″>240,400</property>    </style>   </interface>   <template id = ″t1_switch2″priority = ″0″>     <part>        <part class=″G:Button″ id=″switch2″/>    <style>     <propertyname=″image-src″>AIR1_resorce_switch2.jpg</property>     <propertyname=″g:size″>30,30</property>     <propertyname=″g:text″>Power</property>   </style>   <behavior>      <rule>      <condition>         <op name=″and″>          <eventclass=″KeyListener.g:keypressed″ />          <op name=″egual″>          <property event-class=″KeyListener.g:keypressed″name=″keyCode″ />           <constant value=″57398″ />          </op>       </op>       </condition>       <action>        <callname=″AIR1App.on″/>       </action>       </rule>   </behavior>   </part>  </template>  <template id = ''t1_wind_up″priority = ″5″ relate_id=″wind_1″>    <part>       <partclass=″G:Button″ id=″wind_up″/>    <style>    <propertypart-name=″wind_up″ name=″image-src″>AIR1_resorce_wind_up.jpg</property>      <property name=″g:size″>30,20</property>       <propertyname=″g:text″>Wind Direction +</property>   </style>   <behavior>   <rule>       <condition>        <op name=″and″>          <eventclass=″KeyListener.g:keypressed″ />          <op name=″equal″>          <property event-class=″KeyListener.g:keypressed″name=″keyCode″ />           <constant value=″57399″ />          </op>       </op>       </condition>       <action>        <callname=″AIR1App.wind_up″/>       </action>       </rule>   </behavior>   </part>  </template>  <template id =″t1_wind_down″priority = ″5″ relate_id=″wind_1″>    <part>       <partclass=″G:Button″ id=″wind_down″/>   <style>   <propertyname=″image-arc″>AIR1_resorce_wind_down.jpg</property>       <propertyname=″g:size″>30,20</property>       <property name=″g:text″>WindDirection−</property>  </style>  <behavior>   <rule>    <condition>    <op name=″and″>      <event class=″KeyListener.g:keypressed″ />     <op name=″equal″>       <propertyevent-class=″KeyListener.g:keypressed″ name=″keyCode″ />       <constantvalue= ″57340″ />      </op>     </op>    </condition>    <action>    <call name=″AIR1App.wind_down″/>    </action>    </rule> </behavior>   </part> </template> <template id = ″t1_set_up″priority = ″1″ relate_id=″set_1″>   <part>    <part class=″G:Button″id=″set_up″/>   <style>   <propertyname=″image-src″>AIR1_resorce_set_up.jpg</property>   <propertyname=″g:size″>20,30</property>   <property name=″g:text″>Temperature+</property>  </style>  <behavior>   <rule>    <condition>     <opname=″and″>      <event class=″KeyListener.g:keypressed″ />      <opname=″equal″>       <property event-class″' KeyListener.g:keypressed″name=″keyCode″ />       <constant value=″57341″ />      </op>     </op>   </condition>    <action>     <call name=″AIR1App.set_up″/>   </action>    </rule>  </behavior>   </part> </template> <template id= ″t1_set_down″ priority = ″1″ relate_id=″set_1″>   <part>    <partclass=″G:Button″ id=″set_down″/>   <style>   <propertyname=″image-src″>AIR1_resorce_set_down.jpg</property>   <propertyname=″g:size″>20,30</property>   <property name=″g:text″>Temperature−</property>  </style>  <behavior>   <rule>    <condition>     <opname=″and″>      <event class=″KeyListener.g:keypressed″/>      <opname=″equal″>       <property event-class=″KeyListenerg:keypressed″name=″keyCode″ />       <constant value=″57342″ />      </op>     </op>   </condition>    <action>     <call name=″AIR1App.set_down″/>   </action>    </rule>  </behavior>   </part> </template> <template id= ″t1_set_window″ priority = ″0″>   <part>    <part class=″G:Button″id=″set_window″/>   <style>   <propertyname=″image-src″>AIR1_resorce_set_window.jpg</property>   <propertyname=″g:size″>20,30</property>   <property name=″g:text″>PresetTemperature</property>   </style>   <behavior>    <rule>     <condition>     <op name=″and″>       <event class=″KeyListener.g:keypressed″/>      <op name=″equal″>        <propertyevent-class=″KeyListener.g:keypressed″ name=″keyCode″ />       <constant value=″57343″ />       </op>      </op>    </condition>     <action>      <call name=″AIR1App.set_window″/>    </action>     </rule>   </behavior>    </part>  </template> <template id = ″t1_mode_change″ priority = ″1″>    <part>     <partclass=″G:Button″ id=″mode″/>    <style>    <propertyname=″image-src″>AIR1_resorce_mode.jpg</property>    <propertyname=″g:size″>20,30</property>   </style>   <behavior>    <rule>    <condition>      <op name=″and″>       <eventclass=″KeyListener.g:keypressed″ />       <op name=″equal″>       <property event-class=″KeyListener.g:keypressed″ name=″keyCode″/>        <constant value=″57344″ />       </op>      </op>    </condition>     <action>      <call name=″AIR1App.mode_change″/>    </action>     </rule>   </behavior>    </part>  </template> <template id = ″t1_menu″ priority = ″5″>    <part>     <partclass=″G:Button″ id=″menu″/>    <style>    <propertyname=″image-src″>AIR1_resorce_menu.jpg</property>    <propertyname=″g:size″>20,30</property>   </style>   <behavior>    <rule>   <condition>     <op name=″and″>      <eventclass=″KeyListener.g:keypressed″ />      <op name=″equal″>      <property event-class=″KeyListener.g:keypressed″ name=″keyCode″ />      <constant value=″57345″ />      </op>     </op>    </condition>   <action>     <call name=″AIR1App.menu″/>    </action>    </rule>  </behavior>    </part>  </template>  <peers>    <presentationbase=″XXX.uiml″ />    <logic id =″log1″ source = ″#t2_logic″>   </logic>  </peers> </uiml>

Next, the following is an exemplary light1_interface.uiml file as the UIdefinition file constituting a lighting remocon UI.

<?xml version=″1.0″ encoding=″Shift_JIS″?> <uiml>  <interface id =″light1_interface″ particular name = ″Lighting 1″>     <structure>       <part class=″G:TopContainer″ id=″top″>        <partclass=″G:Image″ id=″bg″/>        <part class=″VBOX″ id=″vbox1″>        <part class=″G:Area″ id=″a1″ source=″#t1_switch3″ how=″union″/>        <part class=″HBOX″ id=″hbox1″>           <part class=″G:Area″id=″a2″ source=″#t1_mame_den″ how=″replace″/>           <partclass=″G:Area″ id=″a3″ source=″#t1_timer '' how=″replace″/>        </part>        </part>        </part>     </structure>    <style>        <property part-name=″bg″name=″image-src″>TV_resorce_bg.jpg</property>        <propertypart-name=″header″ name=″g:location″>0,0</property>        <propertypart-mame=″header″ name=″g:size″>240,400</property>     </style>    <behavior>     </behavior>  </interface>  <template id = ″t1_switch3″ priority = ″0″>     <part>        <part class=″G:Button″id=″switch3″/>     <style>     <propertyname=″image-src″>Light1_resorce_switch3.jpg</property>     <propertyname=″g:text″>Power</property>       <property part-name=″switch1″name=″g:size″>30,30</property>   </style>   <behavior>      <rule>      <condition>        <op name=″and″>          <eventclass=″KeyListener.g:keypressed″/>          <op name=″equal″>           <property event-class=″KeyListener.g:keypressed″name=″keyCode″ />            <constant value=″57398″ />          </op>       </op>       </condition>       <action>        <callname=″Light1App.on″/>       </action>       </rule>   </behavior>   </part> </template> <template id = t1_mame_den ″/>    <part>      <part class=″G:Button″ id=″mame_den″/>    <style>    <propertyname=″image-src″>Light1_resorce_mameden.jpg</property>    <propertyname=″g:text″>Miniature Bulb</property>       <propertypart-name=″select_up″ name=″g:size″>30,20</property>   </style>  <behavior>    <rule>       <condition>        <op name=″and″>         <event class=″KeyListener.g:keypressed″/>          <opname=″equal″>            <propertyevent-class=″KeyListener.g:keypressed″ name=″keyCode″ />           <constant value=″57399″ />          </op>        </op>      </condition>       <action>        <call name=″ Light1App.mame″/>      </action>       </rule>   </behavior>    </part>  </template><template id = ″t1_select_down″priority = ″1″ relate_id=″switch3″>   <part>       <part class=″G:Button″ id=″timer″/>    <style>   <property name=″image-src″>TV_resorce_select_down.jpg</property>   <property name=″g:size″>30,20</property>    <propertyname=″g:text″>Timer OFF</property>   </style>   <behavior>    <rule>    <condition>      <op name=″and″>       <eventclass=″KeyListener.g:keypressed>       <op name=″equal″>       <property event-class=″KeyListener.g:keypressed″ name=″keyCode″/>        <constant value=″57340″ />       </op>      </op>    </condition>     <action>      <call name=″TVApp.select_down″/>    </action>     </rule>   </behavior>    </part>  </template>  <peers>   <presentation base=″XXX.uiml″ />    <logic id =″log1″ source =″#t1_logic″>    </logic>  <peers> </uiml>

Furthermore, the following is an extracted part of an exemplarycomp1.uiml file as the compound UI definition file defining a compoundUI, composed of the selected UI objects, generated based on the UIobject definition information of the UI object selected from a pluralityof UIs, as described with reference to FIG. 3.

<?xml version=″1.0″ encoding=″Shift_JIS″?> <uiml>  <interface id =″comp1_interface″ particular name = ″Compound UI 1″>   <structure>    <part class=″G:TopContainer″ id=″top″>     <part class=″G:Image″id=″bg″/>     <part class=″VBOX″ id=″vbox1″>      <part class=″HBOX″id=″hbox1″>       <part class=″G:Area″ id=″a1″ source=″#t1_comp1(1)″how=″union ″/>       <part class=″G:Area″ id=″a2″ source=″#t1_comp1(2)″how=″ union ″/>      </part>      <part class=″HBOX″ id=″hbox2″>      <part class=″G:Area″ id=″a3″ source=″#t1_comp1(3)″ how=″ union ″/>     </part>      <part class=″HBOX″ id=″hbox3″>       <partclass=″G:Area″ id=″a4″ source=″#t1_comp1(4)″ how=″union″/>       <partclass=″G:Area″ id=″a5″ source=″#t1_comp1(5)″ how=″union″/>       <partclass=″G:Area″ id=″a6″ source=″#t1_comp1(6)″ how=″union″/>      </part>     <part class=″HBOX″ id=″hbox4″>       <part class=″G:Area″ id=″a7″source=″#t1_comp1(7)″ how=″union″/>       <part class=″G:Area″ id=″a8″source=″#t1_comp1(8)″ how=″union″/>       <part class=″G:Area″ id=″a9″source=″#t1_comp1(9)″ how=″union″/>      </part>     </part>     </part>  </structure>   <style>     <property part-name=″bg″name=″image-src″>comp_commonbg.jpg</property>     <propertypart-name=″header″ name=″g:location″>0,0</property>     <propertypart-name=″header″ name=″g:size″>240,400</property>   </style> </interface>  <template id = ″ t1_comp1(1)″ >   <part>    <partclass=″G:Button″ id=″switch2″particular_name= Air Conditioner Manufactured By X″/>  <style> <property part-name=″t1_comp1(1)_switch2″name=″image-src″>AIR1_resorce_switch2.jpg</property>  <propertypart-name=″t1_comp1(1)_switch2″ name=″g:size″>30,30</property> <property part-name=″t1_comp1(1)_switch2″ name=g:text″>Power</property> </style>  <behavior>     <rule>       <condition>           <opname=″and″>            <event class=″KeyListener.g:keypressed″ />           <op name=″equal″>             <propertyevent-class=″KeyListener.g:keypressed″ name=″keyCode″ />            <constant value=″57398″ />            </op>           </op>      </condition>       <action>           <call name=″AIR1App.on″/>      </action>       </rule>  </behavior>   </part> </template><template id = t1_comp1(2)/>   <part>         <part class=″G:Button″id=″switch 1″ particular_name= TV1″/>    <style>    <propertypart-name=″t1_comp1(2)_switch2name=″image-src″>TV_resorce_switch1.jpg</property>    <propertypart-name=″t1_comp1(2)_switch2″ name=g:text″>Power</property>   <property part-name=″t1_comp1(2)_switch2name=″g:size″>30,30</property>   </style>   <behavior>     <rule>       <condition>        <op name=″and″>           <eventclass=″KeyListener.g:keypressed″ part-name=″t1_compl(2)_switch2/ >          <op name=″equal″>            <propertyevent-class=″KeyListener.g:keypressed″ name=″keyCode″ />           <constant value=″57398″ />           </op>          </op>     </condition>      <action>          <call name=″TVApp.on″/>     </action>          </rule>     </behavior>      </part>  </template>              (The rest is omitted...)   <peers>     <presentation base=″XXX.uiml″ />      <logic id =″log1″ source =″#t1_logic″>      </logic>   </peers> </uiml>

INDUSTRIAL APPLICABILITY

According to the present invention, in order to compound a plurality ofUIs, it is possible to generate a single compound UI which adopts onlyselected UI objects and fits in a predetermined UI display area.Accordingly, the compound UI is composed of only UI objects desired bythe user, omitting all unnecessary UI objects. The compound UI generatedin this manner enables concurrent use of the plurality of UIs withoutswitching therebetween. Moreover, unlike multi-window, since there is noneed of operations to select the display area (window) and move a scrollbar in the display area, it significantly reduces operational load forthe user.

1. An electronic device comprising: a display configured to display ascreen for selecting a user interface object for remotely controllingone or more predetermined apparatuses; a controller configured togenerate a remote control screen customized by using the user interfaceobject selected on the screen for selecting; wherein the displaydisplays the generated remote control screen.
 2. An electronic deviceaccording to claim 1, wherein at least two of the predeterminedapparatuses are of mutually different kinds.
 3. An electronic deviceaccording to claim 2, wherein the apparatuses of different kinds includeat least any one of a television, an air conditioner, a lightning, andan audio.
 4. An electronic device according to claim 1, wherein thedisplay enable to display the user interface object on the screen forselecting for each the predetermined apparatus.
 5. An electronic deviceaccording to claim 1, wherein the controller moves a displayed positionof the user interface object if the selected user interface object isdisplayed.
 6. An electronic device according to claim 1, wherein thedisplay displays the selected user interface object with displaysindicating the predetermined apparatus.
 7. An displaying methodincluding: displaying a screen for selecting a user interface object forremotely controlling one or more predetermined apparatuses; generating aremote control screen customized by using the user interface objectselected on the screen for selecting; and displaying the generatedremote control screen.
 8. Non-transitory computer-readable storagemedium storing a program for causing a computer to execute a process,comprising: displaying a screen for selecting a user interface objectfor remotely controlling one or more predetermined apparatuses;generating a remote control screen customized by using the userinterface object selected on the screen for selecting; and displayingthe generated remote control screen.